This report was generated on 2017-11-27 12:16:49.
…
The preprocessing and analysis of the data was conducted in the R project for statistical computing. The RMarkdown script used to generate this document and all the resulting data can be downloaded under this link. Through executing main.Rmd, the herein described process can be reproduced and this document can be generated. In the course of this, data from the folder ìnput will be processed and results will be written to output.
The code for the herein described process can also be freely downloaded from https://github.com/mycid/rddj-template-TE-Raster-Extract.
The code for the project template may be found at https://github.com/grssnbchr/rddj-template.
for creating this extremely elegant and useful template …
This project was created to facilitate easy conversion of L2 netCDF rasters taken from NASA’s OceanColor Website run by the Ocean Biology Processing Group (OBPG) at NASA’s Goddard Space Flight Center (https://oceancolor.gsfc.nasa.gov/). Potential exists for others to easily adapt this script to handle L1 netCDF files as well or other large netCDF remote sensing files. The goals of the code are two-fold: 1. Convert many unprojected netCDF rasters to any raster file type in bulk. 2. Reproject and clip to the same study area many netCDF rasters with widely different swatch bounding boxes. An additional step has been to quickly visualize the results when necissary.
… ### Data description of input files Input files are netCDF files downloaded from the Ocean Color website as L2 products. These products are processed variables such as seasurface temperature and chlorophyll which are dervived from imperical relationships between irradiance and the physical variables. The l2 product is conveniently defined here: https://oceancolor.gsfc.nasa.gov/products/ netCDF files are structured to contain global attirbutes, band attributes and multiple bands. The metadata and file naming system is described here: https://oceancolor.gsfc.nasa.gov/docs/format/l2nc/ Example data for running this script includes 3 files containing chlorophyll alpha concentrations with associated variables and 3 files containing sea surface temperature with associated variables. Both files correspond to three images the study area contains on a single day, December 28, 2016.
NASA_proj_metaDContains essential metadata extracted from the netcdf file global atributes. The bands extracted and the output raster file name are appended in columns at the end of the table. The metadata descriptive document may be found at: FILL IN
Output Raster filesOutput raster files are generated and named using user specification of the function OCnetCDF2Raster. Raster files will be bounded by the specified study area and in the specified projection. Raster will not be scaled and cell values will contain the original values from the extracted raster. During rasterization and reprojection cells with multiple values from the netcdf file are averaged. NASA netCDF files are unprojected and use the WGS1984 latlon coordinate system. Each cell is given a centroid coordinate. Each raster has a resolution of aproximately 1km. This project uses UTM_S18 for the projection. Cells without data are NA. Users have the option of extracting the flagging layer contained within the netcdf file. For convenience, the flag values are described at the end of the document.
Output plots (optional)Output plots can optionally be generated using the ExtractRaster or OCnetCDF2Raster function. Plots are saved as jpeg imaged and display the netCDF raster data in 10 discreet colored quantiles. The study area extent is also shown as a white polygon rectangle border. All points outside of the polygon are not included in the exported raster file but are shown here in the plot. Plots are titled by the date of the swath and saved with the same name as the original netcdf file included. …
# from https://mran.revolutionanalytics.com/web/packages/checkpoint/vignettes/using-checkpoint-with-knitr.html
# if you don't need a package, remove it from here (commenting is probably not sufficient)
# tidyverse: see https://blog.rstudio.org/2016/09/15/tidyverse-1-0-0/
cat("
library(raster)
library(ncdf4)
library(ggmap)
library(svMisc)
library(RColorBrewer)
library(tidyverse) # ggplot2, dplyr, tidyr, readr, purrr, tibble
library(stringr) # string manipulation
library(scales) # scales for ggplot2
library(jsonlite) # json
library(lintr) # code linting
library(sp) # spatial data handling
library(rgeos) # spatial data handling
library(rgdal) # spatial data handling",
file = "manifest.R")
# if checkpoint is not yet installed, install it (for people using this
# system for the first time)
if (!require(checkpoint)) {
if (!require(devtools)) {
install.packages("devtools", repos = "http://cran.us.r-project.org")
require(devtools)
}
devtools::install_github("checkpoint",
username = "RevolutionAnalytics",
ref = "v0.3.2", # could be adapted later,
# as of now (beginning of July 2017
# this is the current release on CRAN)
repos = "http://cran.us.r-project.org")
require(checkpoint)
}
# nolint start
if (!dir.exists("~/.checkpoint")) {
dir.create("~/.checkpoint")
}
# nolint end
# install packages for the specified CRAN snapshot date
checkpoint(snapshotDate = package_date,
project = path_to_wd,
verbose = T,
scanForPackages = T,
use.knitr = F)
rm(package_date)
source("manifest.R")
unlink("manifest.R")
sessionInfo()
## R version 3.4.2 (2017-09-28)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 7 x64 (build 7601) Service Pack 1
##
## Matrix products: default
##
## locale:
## [1] LC_COLLATE=English_United States.1252
## [2] LC_CTYPE=English_United States.1252
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United States.1252
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] rgdal_1.2-7 rgeos_0.3-23 lintr_1.0.0
## [4] jsonlite_1.4 scales_0.4.1 stringr_1.2.0
## [7] dplyr_0.5.0 purrr_0.2.2.2 readr_1.1.1
## [10] tidyr_0.6.3 tibble_1.3.3 tidyverse_1.1.1
## [13] RColorBrewer_1.1-2 svMisc_0.9-70 ggmap_2.6.1
## [16] ggplot2_2.2.1 ncdf4_1.16 raster_2.5-8
## [19] sp_1.2-4 checkpoint_0.4.2
##
## loaded via a namespace (and not attached):
## [1] reshape2_1.4.2 haven_1.0.0 lattice_0.20-35
## [4] colorspace_1.3-2 htmltools_0.3.6 yaml_2.1.14
## [7] rlang_0.1.1 foreign_0.8-69 DBI_0.6-1
## [10] readxl_1.0.0 modelr_0.1.0 jpeg_0.1-8
## [13] plyr_1.8.4 cellranger_1.1.0 munsell_0.4.3
## [16] gtable_0.2.0 rvest_0.3.2 RgoogleMaps_1.4.1
## [19] mapproj_1.2-4 psych_1.7.5 evaluate_0.10.1
## [22] knitr_1.17 forcats_0.2.0 rex_1.1.1
## [25] parallel_3.4.2 broom_0.4.2 proto_1.0.0
## [28] Rcpp_0.12.13 geosphere_1.5-5 backports_1.1.1
## [31] mnormt_1.5-5 rjson_0.2.15 hms_0.3
## [34] png_0.1-7 digest_0.6.12 stringi_1.1.5
## [37] grid_3.4.2 rprojroot_1.2 tools_3.4.2
## [40] magrittr_1.5 maps_3.1.1 lazyeval_0.2.0
## [43] xml2_1.1.1 lubridate_1.6.0 assertthat_0.2.0
## [46] rmarkdown_1.8 httr_1.2.1 R6_2.2.1
## [49] nlme_3.1-131 compiler_3.4.2
# if you want to outsource logic to other script files, see README for
# further information
knitr::read_chunk("scripts/Extract_Rasterize.R")
source("scripts/Extract_Rasterize.R")
paths <- list.dirs(path = "input/")
files <- list.files(path = paths, pattern = ".nc$", full.names = TRUE)
if (length(files) == 0){
message("input netCDF files are not present in the input folder")
} else{
message(paste(length(files), "files are present"))
}
## 6 files are present
#Choose the band you wish to extract
band <- "chlor_a"
#name of netCDF file
nCDF <- files[1]
bnd <- paste("geophysical_data/", band, sep = "") # create full length band variable name
md <- metaXtract(nCDF, band = bnd)
print(md)
## title product_name platform
## 1 HMODISA Level-2 Data A2016363183500.L2_LAC_OC.nc Aqua
## id date_created
## 1 2014.0.1/L2/A2016363183500.L2_LAC_OC.nc 2017-01-12T21:37:07.000Z
## spatialResolution geospatial_lat_max geospatial_lat_min
## 1 1 km -43.70087 -59.26575
## geospatial_lon_max geospatial_lon_min data1 data2
## 1 -59.34376 -81.40116 geophysical_data/chlor_a flags
## file
## 1 input//netCDF_files/A2016363183500.L2_LAC_OC.x.nc
#Set additional variables
output.proj <- "+init=epsg:5362"
date <- NetCDFdateT(nCDF)
plotname <- paste(band, date[1])
proj.ext <-
extent(250000,
xmax = 1200000,
ymin = 3700000,
ymax = 5500000)
#
Rasta <- ExtractRaster(
nCDF,
band = band,
Noflags = FALSE,
output.proj = output.proj,
proj.Ext = proj.ext,
makeplots = TRUE,
saveplots = TRUE,
plotname = plotname
)
## [1] "input//netCDF_files/A2016363183500.L2_LAC_OC.x.nc"
####Check output folder to verify results
#Set stage for use of the netCDF2Raster function
paths <- list.dirs(path = "input/")
files.oc <- list.files(path = paths, pattern = "OC\\.x\\.nc$", full.names = TRUE)
files.sst <- list.files( pattern = "SST\\.x\\.nc$", full.names = TRUE)
#list
chlor <- "chlor_a"
sst <- "sst"
proj.ext <-
extent(250000,
xmax = 1200000,
ymin = 3700000,
ymax = 5500000)
output.proj <- "+init=epsg:5362"
oc.netCDF.2raster(files.oc,
chlor,
filename = NA,
output.extension = ".tiff",
output.proj = output.proj,
meta.name = "NASA_proj_metaD_Chlor.csv",
proj.Ext =proj.ext,
meta.exp = TRUE,
Noflags = FALSE,
makeplots = TRUE,
saveplots = FALSE,
makeMeta = TRUE)
## [1] "Extracting metadata..."
## Progress: 33% Progress: 67% Progress: 100% Done!
## [1] "Extracting Rasters..."
## [1] "input//netCDF_files/A2016363183500.L2_LAC_OC.x.nc"
## Progress: 33% [1] "input//netCDF_files/A2016363184000.L2_LAC_OC.x.nc"
## Progress: 67% [1] "input//netCDF_files/A2016363201500.L2_LAC_OC.x.nc"
## Progress: 100% Done!
## user system elapsed
## -93.63 -24.50 -149.03
The code in this RMarkdown is listed with the lintr package, which is based on the tidyverse style guide.
lintr::lint("main.Rmd")
## main.Rmd:100:64: style: Trailing whitespace is superfluous.
## path_to_wd <- csf()# if this - for some reason - does not work,
## ^
## main.Rmd:200:1: style: lines should not be more than 80 characters.
## bnd <- paste("geophysical_data/", band, sep = "") # create full length band variable name
## ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## main.Rmd:233:1: style: lines should not be more than 80 characters.
## files.oc <- list.files(path = paths, pattern = "OC\\.x\\.nc$", full.names = TRUE)
## ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
## main.Rmd:251:21: style: Put spaces around all infix operators.
## proj.Ext =proj.ext,
## ^~
Flags and meaning long_name: Level-2 Processing Flags valid_min: NA valid_max: 2147483647 flag_masks: 1 flag_masks: 2 flag_masks: 4 flag_masks: 8 flag_masks: 16 flag_masks: 32 flag_masks: 64 flag_masks: 128 flag_masks: 256 flag_masks: 512 flag_masks: 1024 flag_masks: 2048 flag_masks: 4096 flag_masks: 8192 flag_masks: 16384 flag_masks: 32768 flag_masks: 65536 flag_masks: 131072 flag_masks: 262144 flag_masks: 524288 flag_masks: 1048576 flag_masks: 2097152 flag_masks: 4194304 flag_masks: 8388608 flag_masks: 16777216 flag_masks: 33554432 flag_masks: 67108864 flag_masks: 134217728 flag_masks: 268435456 flag_masks: 536870912 flag_masks: 1073741824 flag_masks: NA flag_meanings: ATMFAIL LAND PRODWARN HIGLINT HILT HISATZEN COASTZ SPARE STRAYLIGHT CLDICE COCCOLITH TURBIDW HISOLZEN SPARE LOWLW CHLFAIL NAVWARN ABSAER SPARE MAXAERITER MODGLINT CHLWARN ATMWARN SPARE SEAICE NAVFAIL FILTER SPARE BOWTIEDEL HIPOL PRODFAIL SPARE